Prostate-specific membrane antigen (PSMA), also known as glutamate carboxypeptidase II (GCPII), is a suitable target for specific delivery of antitumor drugs and diagnostic agents due to its overexpression in prostate cancer cells. In the current work, we describe the design, synthesis, and biological evaluation of novel low-molecular PSMA ligands and conjugates with fluorescent dyes FAM-5, SulfoCy5, and SulfoCy7. In vitro evaluation of synthesized PSMA ligands on the activity of PSMA shows that the addition of aromatic amino acids into a linker structure leads to a significant increase in inhibition. The conjugates of the most potent ligand with FAM-5 as well as SulfoCy5 demonstrated high affinities to PSMA-expressing tumor cells in vitro. In vivo biodistribution in 22Rv1 xenografts in Balb/c nude mice of PSMA-SulfoCy5 and PSMA-SulfoCy7 conjugates with a novel PSMA ligand demonstrated good visualization of PSMA-expressing tumors. Also, the conjugate PSMA-SulfoCy7 demonstrated the absence of any explicit toxicity up to 87.9 mg/kg.